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ABSTRACT 


The quality of software management in a development program is a major factor 
in determining the success of a program. The four main areas where a software program 
manager can affect the outcome of a program are requirements management, 
estimation/planning management, people management, and risk management. In this 
thesis a quality management metric (QMM) was used to measure the performance of ten 
software managers on Department of Defense (DoD) software development programs. 
Informal verification and validation of the metric compared the QMM score to an overall 
program success score for the entire program and yielded positive correlation. The 
results of applying the QMM can be used to characterize the quality of software 
management and can serve as a template to improve software management performance. 
Future work includes further refining the QMM, applying the QMM scores to provide 
feedback and appropriate training to program managers, and using the QMM scores as an 
input to program cost and schedule estimation methodologies to provide better program 


estimates. 
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EXECUTIVE SUMMARY 


The quality of management in a software-development program is a major factor 
in determining the success of the program. Software development by its very nature can 
be a complex process because the product or products developed are logical, rather than 
physical systems, and therefore the development can not be approached as a 
manufacturing problem. Software is not a visible and tangible product. The success or 
failure of a software-development program depends to some degree on the ability of the 
program manager to estimate and plan the program, coordinate the work force efforts, 
facilitate communications, define requirements, and contro] the amount of risk exposure 


the program experiences. 


Program-management tools have been developed to assist the program manager in 
estimating the cost and schedule of software programs. However, the estimation tools 
available assume consistent and high-quality program management. Although the quality 
of software development management tools has improved over the past thirty years, 
software-development programs are still often notoriously behind schedule and over 
budget. Hence there still appears to be something missing in the software-program 
management equation that has been overlooked in the quest to produce cost-effective, 
high-quality software. The missing something may be good management. If the quality 
of the software program management were measurable and available to be input into the 


costing and scheduling tools, then the resulting estimates could be provided to program 
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managers so that they could pinpoint areas of software-program management in which 


improvement needs to be made. 


The Quality Management Metric (QMM) is a survey instrument which measures 
in a repeatable manner, the quality of management for a software-development program. 
The QMM consists of four separate metrics which measure the program manager’s 
performance in the areas of requirements management, estimation/planning management, 
people management, and risk management. The results of the four metrics are then 
combined to provide a Quality Management Metric score which predicts the quality of 
the program management and the success of the program. The QMM survey instrument 
has been applied to measure the performance of three software managers on Department 
of Defense (DoD) software development programs. The resulting QMM scores of the 
program managers were informally verified and validated in a comparison with an overall 


program success score and yielded positive correlation. 


This thesis evaluates the QMM as a viable software-management-quality survey 
instrument. for monitoring, evaluating, and improving the enactment of the software 


process. The survey and this research are a step in that direction. 


In the current research, the QMM survey instrument is used to measure the 
performance of ten program managers on DoD software-development programs. In order 
to gain a measure of the validity of the QMM, the QMM survey instrument was given to 
the program manager and also to one or two other independent individuals who were 
closely associated with the program and knowledgeable about the overall practices and 


success of the program. The program manager QMM scores were compared to the 
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program manager’s subjective overall program success scores and the correlation yielded 
strong positive results. In addition to the comparison of the QMM score with the overall 
program scores, the results from the four separate metrics were analyzed. The study 
examined the correlations between the program manager scores and each sub-area metric 
and the QMM scores and each sub-area metric to determine whether there were 
relationships between the overall program manager performance and the program 
manager performance in a specific area or areas. The independent QMM scores were 
compared with the independent overall program scores and yielded a strong positive 
correlation. The independent scores in each of the four sub-areas were also compared 
against the independent QMM and overall program scores to ieee whether there 
were any relationships between the overall program manager performance as rated by the 


independent and the overall program success, as rated by the independent. 


From the results of the data analysis several conclusions are drawn: 


e The QMM survey instrument yields a strong correlation with the QMM score and 
the overall program score for both the program manager and the independent data 
sets. This indicates the QMM survey instrument does measure the quality of 

_ management on a software-development program where the management policies 
and procedures are the same as or similar to the currently accepted software 
engineering management practices, as defined by the QMM survey instrument. In 
the case where the management practices and procedures are known to diverge 
from the currently accepted practices, the QMM survey instrument may be used 
to measure the level to which the currently accepted practices are implemented, 
but will likely yield a lower QMM score and probably predict a lower success rate 
for the program than the actual rate. 


e The data indicates that, in general, program managers believe that they are good 
managers of people. However, the independent evaluators do not believe the 
program managers are as good at managing people as the program managers 
themselves. 
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e The QMM survey instrument may be useful in detecting discrepancies between 
the program manager’s and the individual team member’s perspectives and 
understanding. 


e The results of the study indicate that there is a great deal of uncertainty involved 
in managing estimation/planning and risk, and that the program managers who are 
good at managing the uncertainty in one of these areas are usually good at 
managing the uncertainty in the other area as well. 

e The data correlations between the program manager data set and the independent 
data set indicate that perhaps the area of requirements management is more visible 


and better understood by the entire development team than other areas of 
management. 


Recommendations for future work include updating and further testing the QMM 
survey instrument, developing a mechanism to provide the program managers with 
feedback on their management performance and possible courses of action to take to 
improve their performance, and integrating the QMM survey instrument results into 


existing cost, schedule and risk models to improve program estimation accuracy. 


XVI 
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I. INTRODUCTION AND BACKGROUND 


A. SOFTWARE DEVELOPMENT 


In addition to the normal development difficulties of a non-software product 
development, software development introduces additional complexity and potential 
problems which with the program manager must deal. Software is a logical, ine: than a 
physical system, therefore, development cannot be approached as a manufacturing 
problem. Software is not a visible and tangible product. Per dollar spent, software 
products are more complex than other engineering products [Ref. 1]. Software is flexible 
and can be easily changed, at least on the surface. Often one of the difficulties facing a 
program manager is the idea that software is flexible and easily changed at any point in 
the development process. Studies have shown that changing software requirements late 
in a development effort has a much more significant and adverse impact on the program 
cost and schedule than changing the requirements early on in the program. However, 
since the development effort is for a software product the customer often asks for changes 
in the requirements up until the end of the program. The program manager must control 
the amount of change to the requirements and communicate the impact of proposed 
changes to the customer to coordinate and maintain control of the change process and 
keep the program on schedule, within budget and produce a quality product the customer 


enjoys. 


The program manager is responsible for gathering the requirements and ensuring 
they fulfill the customer’s needs and wants. Getting the correct requirements is difficult 


because the customer often does not know exactly what they want until they have seen 


] 














the software product or a prototype of the product and try to use it. Many requirements 
are implicit in the customer’s understanding and not made explicit until the customer 
realizes they are missing during the use of the product. There are also misinterpretations 
between the way the customers expect the requirements to be implemented and the way 


the developers interpret them. 


The software development personnel the program manager leads are intelligent, 
educated professionals who demand good management to be highly productive. If they 
believe they are being mismanaged they may rebel against the system. The developers 
thrive on challenging tasks which are perceived as being significant, appropriate feedback 


and peer recognition, and require a good work environment to be their most productive. 


The software development process is further complicated by the interactions and 
communication required for a successful software development program. The program 
manager is responsible for facilitating the flow of communication not only within the 
development program but of equal importance, outside of the program with the 
customers. The customers must be kept up to date on the program requirements, cost, 
schedule, problems, and possible tradeoffs so they can provide feedback to the program 
manager allowing the manager to make informed decisions which will lead to a 
successful program product and a satisfied customer. In addition, the program manager 
must ensure good communication between the developers to reduce misunderstandings 
and the need for rework, particularly if part of the product is being developed by one or 


more subcontractors. 














Various program management tools have been developed to assist the program 
manager in estimating the cost and schedule of software programs. The COnstructive 
COst Model [Ref. 2] or COCOMO model is a well known software development cost and 
schedule estimation tool. It is representative of the type of estimation tools available to 
software program managers in that it uses software product metric measures such as the 
estimated number lines of code delivered in the final product as the basis for the program 
cost and schedule estimations throughout the program development. Although the 
quality of the software development management tools has improved over the past thirty 
years, software development programs are still notoriously behind schedule, over budget, 
etc. For example, in March 1999 Constance Fabian-Isaacs and Ed Robinson [Ref. 3] 


wrote, 


Software development projects are notorious for running over budget and 
behind schedule. The Center for Project Management in San Ramon, 
California reports that 99% of commercial software products are not 
completed on time, within budget, or according to specifications, and that 
the average project is underestimated by 285%. With statistics like these, 
it’s no wonder software project managers are looking for ways to improve 
their processes. 


Clearly, there is still something missing in the software program development 
equation which has been overlooked in the quest to produce cost effective, high quality 
software. Randall Jensen suggests the missing something may be good management 
which he states is “the most important productivity and quality driver” [Ref. 4]. In 1981 


Barry Boehm [Ref. 2] wrote, 


Poor management can increase software costs more rapidly than any other 
factor. 














Gerald Weinberg took Barry Boehm’s concept one step further by summarizing 
the four most important cost drivers and the relative importance of each as identified by 


Barry Boehm; see Figure 1. 
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Figure 1. From [Ref. 5] The Four Most Important Cost Drivers on a Software 
Development Program. 


From Figure 1 it 1s obvious that the quality of the program management may be a 
major factor in the success or failure of a software development program. The 
COCOM6O cost and schedule estimation model, and many others like it, assume uniform, 
good management for both the developer and the customer organizations. From the 
frequency with which software development programs experience cost, schedule and 
performance overruns and problems, the assumption of good management may be 
incorrect in many cases. If the quality of the software program management were 
measurable and available to be input into the cost and schedule tools, the resulting 
estimates would provide more accurate information for the program managers and 


possibly improve the overall program performance. 








B. INSTRUMENTS FOR MEASURING THE QUALITY OF SOFTWARE 
MANAGEMENT 


Finding an instrument to measure the quality of software management is difficult. 
There are many instruments available to measure the product metrics which focus on 
individual aspects of the software item (usually volume) under development. There are 
also many instruments available to measure the process metrics which focus on the 
activities involved in software development. Both metrics typify most program 
performance evaluations and ignore any consideration of the quality of software 


management. 


The People Capability Maturity Model (P-CMM) [Ref. 6] provides a framework 
from which the program organization can stabilize their procedures and get repeatable 
results. However, the P-CMM does not provide a specific set of metrics for measuring 
the quality of or the improvement of program management. Instead, the P- 
CMM assists the program organization in defining its approach to measuring the 
performance of the management, hiring managers which conform to the expected 


performance and training its managers to attain the desired performance. 


There are few if any instruments available to measure the quality of the software 
management. One instrument which does measure the quality of software management is 
the Quality Management Metric (QMM) developed by Martin Machniak [Ref. 7]. The 
QMM measures the software program management quality through the use of a survey 
which is given to the program manager. From the program manager’s responses to the 


pair-choice and yes/no questions a profile of the program manager is built up in the areas 











of requirements management, estimation/planning management, people management and 
risk management. From the manager’s score in each of the four areas a total QMM score 
is computed which predicts the manager’s current and future performance as well as the 


program ’s current and future performance. 


To date the QMM has undergone limited testing. Martin Machniak used the 
QMM survey instrument to measure the performance of three software program 
managers on Department of Defense (DoD) software development programs. The 
resulting QMM scores of the program managers were informally verified and validated in 
a comparison with an overall program success score and yielded positive correlation 


[Ref. 7]. 


This thesis evaluates the QMM as a viable software management quality survey 
instrument. The long-term goal is to provide a means for monitoring, evaluating, and 
improving the enactment of the software process. The survey and this research are a step 


in that direction. 


C. SCOPE OF THE RESEARCH 
Since the QMM has been tested in a limited fashion (3 programs), the thesis 


further explores the validity of the QMM in measuring the quality of software managers 
and in predicting the success of programs. The QMM survey instrument was used to 
measure the performance of ten program managers on DoD software development 
programs. To gain a measure of the validity of the QMM, the QMM survey instrument 
was given to the program manager and also to one to two other individuals who were 


Closely associated with the program and knowledgeable about the overall practices and 
6 














success of the program. The program manager QMM scores were compared to the 
program manager overall program success scores and the correlation yielded positive 
results. In addition to the comparison of the QMM score with the overall program scores, 
the results from the four sub-areas of the QMM: requirements management, 
estimation/planning management, people management, and risk management were 
analyzed as well. The study examined the correlations between the program manager 
scores and each sub-area and the QMM scores and each sub-area to determine if there 
were any relationships between overall program manager performance and the program 
manager performance in a specific area or areas. The independent QMM scores were 
compared with the independent overall program scores and yielded a positive correlation. 
The independent scores in the four sub-areas were also compared against the independent 
QMM and overall program scores to determine if there were any relationships between 
the overall program manager performance as rated by the independent and the overall 


program success as rated by the independent. 


From the results of the data analysis several conclusions are drawn, these are 
presented in Chapter V. Recommendations for future work are also presented in Chapter 


V. 
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Il. RELATED WORK 


A. BACKGROUND OF THE QMM 
The QMM is a repeatable set of metrics that can be used to assess the quality of 


software-development management. It can also be used to measure improvement in the 
quality of the management of software development and predict future success levels of 
software-development projects. The definition of sofware development extends to both 
development of new software and maintenance efforts to maintain and extend the existing 
functionality of systems. Application of the QMM outside of the software-development 
domain has not been addressed in the development or testing of the QMM survey 


instrument. 


The QMM survey instrument consists of four eueeores of software-management 
metrics: requirements management, estimation/planning management, people 
management, and risk management. The scores for each category provide an indication 
of the quality of the program manager’s management of a software-development project. 

The program manager is asked to fill out the QMM survey instrument to attain a 
measure of the quality of the management on the software development program. The 
resulting QMM score predicts the program manager’s overall performance and the 
success of the program. Each of the four sections of the QMM has the same number of 
total points possible. The total points for each section are multiplied by its relative 
Importance Coefficient (IC) to yield a weighted score. . After the weighted scores for each 
section are computed, they are summed together to yield the Quality Management Metric 


(QMM) score [Ref. 7]. 














The IC for each section was determined from the relative rankings of importance 
of each of the sections. The IC rankings were determined through interviews with senior 
program managers and focus groups. 

The overall QMM score is made up of the sum of the scores of the each of the 
individual sections multiplied by an IC for that section. The QMM equation is as follows 
[Ref. 7]: 


QMM = 0.92 RqM + 0.67 EPM + 1.86 PM + 0.55 RkKM 


The QMM is the sum of the four components: 
RqM is the requirements management metric 
EPM is the estimation/planning metric 

PM is the people management metric 


RKM is the risk management metric 


The QMM survey instrument is intended to pertain to only one program at a 
specific point in time. Each program manager who participates in the survey is asked to 
answer the survey questions from the perspective of what was happening on a specific 


program at a specific point in time, such as a milestone or delivery. 


It is envisioned that the QMM survey instrument can be used to. measure the 
improvement in a program manager’s performance. The program manager can take the 


survey at different points in time on the same program or on two different programs, and 
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the results of the surveys can be compared to obtain an indication of the improvement of 
the program manager in managing the software-development program. 

The questions used on the QMM survey instrument were gathered from 
periodicals, textbooks, interviews with senior program managers, and focus groups. The 
intent was to be as inclusive as possible with respect to judging the quality of 
management. Appendix A contains the QMM survey instrument forms with the 
weighting for the individual questions. 

Part one of the questionnaire contains pair-choice questions. These questions 
require the participant to choose one of two statements that best describes what was 
happening on the program at the evaluation point in time. The choice does not have to 
match exactly: the participant selects the choice which best describes the program. The 
model for this type of survey instrument is the Myers-Briggs Type Indicator 
questionnaire. Each pair choice represents two differing ideas in an effort to ascertain a 
tendency of the individual. Often the pair choices are repeated with different wording to 
confirm earlier choices and measure the strength of the tendency [Ref. 7]. Each section 
has approximately 70 questions and each section has a total of 70 points for part one of 
the questionnaire. 

Part two of the questionnaire contains questions to be answered with yes, no, or 
not applicable(n/a). This part of the questionnaire is user-friendly for conducting surveys, 
requiring minimum writing. Each question has a point value based upon the relative 
importance of the question in that section. Each section has approximately fifty questions 
and a maximum score of sixty-two points. The entire survey, including both parts for all 


four sections contains 457 questions. 
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The QMM survey instrument was conceived and targeted for high-level program 
managers. High-level program managers are often in charge of more than just a software 
development effort, including system-level design, hardware development, integration 
and testing of an entire system which includes software development as a component of 
the overall system. The high-level program manager is asked to answer the questions as 
the questions pertain to the software development portion of the program in question. 

In order for a program to qualify for participation in a QMM survey, the 
minimum requirement is that a program manager be in charge of at least three software 
developers. The software developers do not have to be working as software coders; they 
can be working in any of the software development areas such as design, requirements 
analysis, documentation, coding, and testing. A requirement of the software developers 
is that they have defined areas of responsibility. These a can be shared with other 
developers, but specific developers are assigned to each area. 

Administration of the survey was conducted such that the participants had no 
information on the point totals or relative weightings of the questions before hand. This 
was done to avoid introducing additional bias into the results of the survey. Each survey 
was given to the participant in a private, one-to-two hour interview. The results were 
manually scored to focus attention on the entire process and de-emphasize focusing on 
only the final Quality Manaseinent Matic (QMM) score [Ref. 7]. In order to obtain 
complete and full disclosure on the surveys, the results were reported anonymously as 
programs A, B, and C. 

Collectively, the measures in the four sections give an objective view of the 


quality of the software management. Thus, two programs scoring equally on product and 
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process metrics can be further measured and compared on the basis of the quality of their 


management. 


B. SOFTWARE MANAGEMENT COMPONENTS OF THE QMM 


The QMM provides a repeatable set of metrics to determine the quality of 
software management, measure the improvement, and predict future siccees levels of 
programs. The QMM survey instrument is made up of four sections: requirements 
management, estimation/planning management, people management, and risk 
management. 


1. Requirements Management 
Requirements management focuses on managing the process of extracting, 


developing, defining, and refining the requirements of a software development program 
[Ref. 7]. The QMM requirements metric is not a product or process metric for 
requirements nor is it intended to be one. Many product and process metrics already exist 
for requirements. The QMM requirements metric is intended to objectively measure the 
quality of the program manager in the area of requirements management. 

Requirements are necessary for software programs because they define the 
product to be produced. Well documented and written requirements unambiguously state 
the capabilities which the software product will have upon completion and form a 
contract between the customer and the developer which defines the scope of the work to 
be accomplished. The program manager is responsible for establishing procedures to 
ensure that the requirements specification is complete, consistent, readable, lacks 


ambiguity, and can be traced to origins. The program manager must also ensure 
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stakeholder involvement in extracting and refining the requirements to make sure the 
software product will satisfy the customer’s needs. 

The program manager is responsible for validating, prioritizing, and conducting 
tradeoff studies on the requirements. Part of managing the customer expectations is 
validating the requirements. The requirements should be scrubbed to be certain they 
reflect the customer needs accurately. Once the requirements are validated, they should 
be prioritized. Every software development program has cost and schedule constraints. 
Often, the requirements which are extracted would extend the development program 
beyond the cost and schedule constraints and so the requirements are prioritized with the 
customer so that when a constraint exists, the highest priority requirements are executed 
and the lower priority requirements are cut. Requirements prioritization goes hand-in- 
hand with requirements tradeoff studies. The customer and the program manager need to 
know their development options to determine the appropriate requirements and the 
priority of the requirements for the software product. 

The program manager is responsible for implementing change control 
management for the requirements. Scope creep, where the requirements expand due to 
new or changed customer requirements or through the developers throwing in nice-to- 
have features, increase the complexity of the product, thereby increasing the cost to test 
and debug the product. The new requirements also eat into the cost and schedule 
constraints of the program, possibly to the point of reducing the number of original 
requirements which can be accomplished. Requirements change control is necessary to 
limit the amount of unauthorized change, and ideally, to assist in managing the 


customer’s expectations. 
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The QMM requirements metric questionnaire evaluates the program manager on 
the formality of the requirements, the establishment of procedures, extraction 
management, change management, and testability of requirements. The questionnaire 
does not seek to determine the quality of judgment on any specific decisions made. The 
thrust of the questions is to establish the structure, if any, laid out by the program 
manager in the area of requirements [Ref. 7]. How formal are the processes? How well 
understood, and implemented are the processes? Who is involved in the extraction 
process and when is it done? How is change handled? Is each requirement testable and 
when is testing considered during the development? 


2. Estimation/Planning Management 
Estimation and planning management focuses on establishing procedures and 


techniques to accurately estimate the costs and schedule of the development effort, and 
from these estimates properly plan the program. Accurate estimates are necessary to 
produce realistic plans which are essential to avoid cost overruns, late deliveries, 
inefficiency, poor reliability, and lack of user acceptance. 

The program manager is responsible for establishing a frame of reference for the 
program estimates and plans as well as an ability to measure against the reference to 
determine program progress. Three types of measures used to assess program progress 
are as follows: product measures, process measures, and resource measures. Product 
measures are based upon the products produced: lines of code, pages of documentation, 
and number of modules completed within each phase of the program. Process measures 
quantify the behavior, strategy, and the execution of the processes used to develop the 


product. Process measures are based on such things as the number of defects found in 
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testing, the number of requirements changes, and the milestones which are met. 
Resource measures are based on the number of labor hours expended, the appropriate 
assignments of personnel, and the proper work environment to maximize productivity. 

The program manager is responsible for implementing the proper procedures and 
structure to track the product, process, and resource metrics throughout the program, and 
ensuring the measures being used will yield the most accurate and useful results for the 
program. 

The QMM estimation/planning metric questionnaire does not seek to advocate a 
specific estimation technique but rather, it seeks to quantify the quality of the 
management of the estimation and planning process. The questions determine whether 
the program manager is performing both initial and follow up estimation and planning. Is 
documentation completed and used? Are currently acceptable software engineering 
Management methods and practices being employed? Is the program manager managing 
the estimation and planning process sufficiently to give confidence to the program 
estimates and plans? 


3. People Management 
The QMM looks at people management from a_ specific software 


development/management perspective. It focuses on the unique qualities and needs of 
people working in a software-development environment. 

The people-management metric focuses on the program manager’s ability to 
allocate human resources appropriately, implement a structure to facilitate 
communication both within and external to the program, and provide leadership to the 


program. People management is the most important component of the QMM. Quality 
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people management increases the productivity of the workforce and increases the 
program’s chance to meet budget allocations and schedules and be successful. 

The program manager is responsible for establishing the organizational culture at 
the program-development level. This includes recruiting, training, allocating tasks, 
building teams, empowering team members, implementing good organization, mentoring, 
establishing a good work environment and reward structure, and promoting good 
communications. The developer’s productivity is more significantly impacted by the 
team work environment than the overall organizational work environment. The program 
manager is responsible for deveionms the proper work environment on the team level to 
promote the maximum individual productivity from team members. The program 
manager is also responsible for fostering open communications within the development 
team (horizontal), with the program manager (vertical), and with the users and 
stakeholders (external) to the extent possible within the overall organizational structure. 
Program manager leadership and mentoring are important in creating and maintaining the 
team work environment and communication infrastructure. The leadership of the 
program manager is important in enabling the program manager to establish good 
practices for managing human resources. 

The people-management section of the QMM is composed of four sub-sections: 
human resources, leadership, communication, and technical competence. The human 
resources section covers human-resource management. The leadership section covers the 
personal leadership skills exhibited and the leadership mentoring provided by the 
program manager. The communication section covers what communication protocols are 


set up for the program organization and used individually by the program manager. The 
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technical competency questions cover the level of competency of the program manager 
within the program on a technical level. 

The QMM People-Management metric questionnaire covers the people- 
management skills the program manager exhibits and the type of organizational and 
communication structure the program manager establishes at the development-team level. 
The program managers are rated on their personal leadership skills and team mentoring. 
The organizational structure set up determines if the program manager creates an adult- 
to-adult or master-servant type of relationship between the development team and the 
manager. The adult-to-adult management style is one in which the manager asks for the 
development team’s inputs before making decisions or empowers the development team 
to make decisions. The master-servant type of management style is one in which the 
manager does not ask for inputs from the development team; rather, the manager dictates 
the decisions and the tasks to the team. The adult-to-adult is more favorable for 
increasing team member job satisfaction, according the focus group feedback. 

The questions also ascertain the communications protocols which the program 
managers establish for program communication as well as the communications formats 
which the program managers use themselves. 


4. Risk Management 
Risk management is the process of identifying, mitigating, and eliminating 


potential problems before they can do result in a loss, such as a efficacy (i.e., 
productivity) loss. 
The QMM examines risk management by looking at the components of risk 


management: risk assessment, risk control, risk communication, and risk avoidance. Risk 
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assessment is comprised of identifying the risks, analyzing the risks, and prioritizing the 
risks. It is important to identify each risk individually so that they can be dealt with on an 
individual basis. The risks are analyzed for their estimated impact to the program if they 
occur, and then are prioritized. The prioritization allows the program manager to 
concentrate the limited program resources on those risks which are the most likely to 
occur and which will result in the largest of the losses to the program if they do occur. 

Risk contro] is comprised of risk planning, risk tracking, and risk resolution. Risk 
control consists of managing the risks to obtain the desired outcomes. Risk planning 
involves determining appropriate actions to mitigate individual risks, prioritizing the 
actions, and integrating them into an executable risk management plan. Risk tracking 
involves tracking the status of the risks and their mitigation actions through metrics. Risk 
resolution is the execution of the risk mitigation plans for each risk. 

Risk communication is the process of exchanging risk information throughout the 
organization at all levels. This activity is critical to the management of risk, allowing the 
program manager to maintain a level of awareness of the risks, their impacts, and the 
success or failure of the risk mitigation strategies. 

_ Risk avoidance is one possible strategy for risk resolution which involves 
avoiding the more risky development strategies or technical challenges. Examples of risk 
avoidance include using a proven compiler and development environment instead of 
using a new and untried compiler and development environment, and deciding to use a 


proven technology component instead of relying upon the development of a new cutting 


edge technology. 














All programs have some element of risk as do all areas of the program— 
requirements management, estimation/planning management, and people management. 
Quality management acknowledges the presence of risk and implements some level of 
systematic risk management. 

The QMM risk management metric measures the risk management structure 
which is set up by the program manager to deal with risks. The questionnaire examines 
the structures used by the program management for identification, monitoring, and 
managing risk. The questions determine whether the program manager has implemented 
strategies and personnel to assess, efor: and prioritize all reasonable risks and also if a 
risk management plan exists and is implemented and updated throughout the program’s 


development. 


c: INITIAL VALIDATION AND VERIFICATION OF THE QMM 
The approach to the verification and validation of the QMM was informal. Three 


DoD programs were evaluated for a QMM score. The program manager and one 
program development team member evaluated the program management on programs A 
and C. On program B, the program manager and two program development team 
members evaluated the program management. 

In order to provide a frame of reference in which to correlate the survey results, 
two other measures were developed and used. These two measures are the QMM 
percentage score and the overall program success score [Ref. 7]. 

The QMM percentage score is a derived measure of the overall QMM score. The 


maximum possible score for the entire survey is 528 points and the minimum possible 
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score is -130.86 as part-two questionnaires contain negative point response values. In 
order to obtain a QMM percentage score, the following steps are taken. First, the survey 
minimum possible score is normalized to zero. Since the survey minimum QMM score is 
—130.86, 130.86 is added to the survey minimum possible score so that it equals zero. 
Correspondingly, 130.86 must be added to both the survey maximum score possible and 
to the actual QMM score obtained in the survey. Since the QMM survey maximum 
possible score is 528.0, the resulting normalized survey maximum possible score is 
658.86 [Ref. 7]. 

To obtain a QMM percentage | the normalized QMM score obtained from 
the survey is divided by the normalized survey maximum possible QMM score and then 


multiplied by 100. The equations are as follows [Ref. 7]: 


QMMmum + 130.86 = 0.00 = QMMnaw NORMALIZED 
QMMnax + 130.86 = 658.86 = QMMmax NORMALIZED 
QMMscors + 130.86 = QMMscore NoRMALIZED 


(QMMscorE NORMALIZED/QMMmax normauzep) * 100 = QMMpercenrAGcE SCORE 


The overall program success score is a subjective number assigned by the survey 
participant rating the overall success of the program. The success of the program is 
measured in terms of how well the final product performs on a scale of zero to ten. On 
this scale zero corresponds to complete and utter failure with no viable product produced. 
A score of ten on the scale is not realizable, except theoretically; it corresponds to 
complete success, no bugs in the product, delivery on time and within budget, and 
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complete customer satisfaction with the end product. The cause for success or failure of 
the program is not important. It may or may not be associated with any actions involving 
program management. 

In addition to the QMM percentage score and the overall program score, a mean 
overall success score was obtained for each of the three programs. The mean overall 
success score is derived from each survey participant’s overall program success score and 
at least two other individuals (mostly users, or those somehow associated with the 
program or delivered product) able to judge the overall success of the program [Ref. 7]. 

The goal underlying the data analysis was to determine any correlation between 
the participant’s QMM score, their individual overall program score, and the mean 
overall success ranking of the program. This was done by comparing the QMM 
percentage score with the individual overall program success score and the mean overall 
success score for the particular program. For example, an overall program success score 
of seven corresponding to a QMM percentage score of seventy percent plus or minus five 
percent indicates a strong correlation. An overall program success score corresponding 
to a QMM percentage score of between plus or minus five and fifteen percentage points 
of the QMM percentage score indicates fair correlation. An overall program success 
score corresponding to more than plus or minus fifteen percentage points of the QMM 
percentage score indicates weak soneladoie If the QMM percentage score has weak 
correlation with the program overall success score, then the QMM metric 1s still valid as 
programs could conceivably fail for a variety of reasons, including a poor choice of 
technology or a funding shortfall. The reverse condition may also be true for explaining 


successful programs with low quality of software management. However, it is typically 
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expected that successful software programs follow superior software program 
management [Ref. 7]. 


The results of the data analysis are shown in Table 1 [Ref. 7]. 





‘SuccessScore | 7 | 7 | 9 | 4 [| 3 | 4 | 4 | 
4 


Mean Success 7 4 
Score (0 — 10) 


Table 1. From [Ref. 7] Results Summary Comparison. 


Program | Program Req. Est./ | People 
Score Man. /|Planning| Man. 
Man. 
A-PM | 


| 9 | 78.4% | 66.5% | 76.6% | 85.6% | 72.9% _ 
3 | (6. 
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6. 
26.9% | 28.4% 
15.6% | 80.6% | 15.1% 


48.5% | 37.4% | 34.7% | 68.1% | 10.8% 


Table 2. Data From the QMM Survey Summary Sheets 


The QMM summary sheets for each survey are summarized in Table 2. The 
program manager for program A is referred to in the tables as A — PM, the independent 
evaluator on program A is referred to as A -— 1, and so on. An error in the math on the 


QMM summary sheets has been corrected in the results presented in Table 2. Therefore, 
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the QMM percentage scores vary slightly between Table 1 and Table 2. Table 2 also 
presents the percentage of the points scored in each of the four sections: requirements 
management, estimation/planning management, people management, and_ risk 
management. 

Programs A and C exhibit correlation between the QMM percentage score and 
both the overall program success score and the mean overall success score. By 
examining Table 2 it can be seen that program A shows a weak risk-management section. 
This appears to be correct, as risk management for this program was not emphasized. 
The higher scores in the other eee reflect the fact that program A was highly 
structured and planned, had successful requirements extraction, and enjoyed good 
technical success with their deliverables. 

Program C had essentially no risk miamasemeat little planning, and poor 
requirements extraction. Once again this is reflected in the scores in Table 2. However, 
this program did have good people-management which allowed them to deliver a usable 
product, the high people management skill level is also reflected in Table 2. 

Program B appears to have a disconnect between the perceptions of the program 
manager and the development team members interviewed. The difference in perception 
is possibly due to a widely documented effect in social psychology in which people 
prefer to see themselves in a self-enhancing fashion. On the job, approximately ninety 
percent of managers and workers rate their performance superior to that of their peers 
[Ref. 7]. 

The conclusion of the thesis was that the goal of creating repeatable metrics for 
determining the quality of software management was obtained. Further work is needed to 
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refine the survey questions in wording and clarity, the point values for the questions need 
to be refined to improve the survey, and additional case studies are required to fine tune 


the QMM metrics. 
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Ill. CURRENT WORK WITH THE QMM 


A. METHODOLOGY AND PROTOCOL 
The goal of the current work with the QMM is to further explore the validity of 


using the QMM for measuring the quality of software program management. 


In this research, the QMM survey instrument is used to measure the performance 
of ten program managers on DoD software development programs. The ten program 
managers are asked to complete the survey instrument. In addition, the survey instrument 
is given to one to two development team members on each program who are 
knowledgeable about the overall practices and success of the program. A requirement for 
choosing the individual development team members is to select team members who have 
a good understanding of the overall program and are knowledgeable about the 
management practices and infrastructure implemented by the program manager 
throughout the program. The intent is to choose individuals whose experience is not 
limited to specific areas of the program. The individual development team members are 
asked to answer the QMM survey questions as they pertain to the program management 
and the management characteristics of the program manager being evaluated. The 
program manager is asked to determine a specific point in time on the program, such as a 
milestone or delivery, for the evaluation of the program management and to define it such 
that the individual development team members will be able to identify with the point in 


time that was selected and evaluate the program for that same point in time. 


The survey instrument is applied to the interviewees in two ways—either via a 


private one-on-one personal interview which lasts approximately two hours, or via an 
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electronic copy which is distributed by email, filled out by the interviewee and returned. 
In eight of the ten programs (17 surveys total) the survey instrument is applied in 
personal interviews, in the remaining two, programs H and J, (four surveys total) the 
survey instrument is applied through email. The emailed surveys executed later in the 
cca process benefited from the experience gained in the execution of the surveys via 
personal interview. By the time three of the four emailed surveys were distributed, a 
short data dictionary of terms on the survey and answers to frequently asked questions 
was available and included with the survey distribution. The questionable terms in the 
data dictionary and frequently asked questions were identified from feedback obtained 
during the in-person interviews. The interviewees surveyed via personal interview, 
programs A — G and program I, benefited from the opportunity to ask for clarification on 


terms and the intent of individual questions during the interview process. 


The format for the personal interviews included scheduling the interview in a 
conference room or a quiet place away from interruptions, executing the survey, and 
asking the participant to subjectively rate the success of the program. The interviewees 
are given the survey instrument on paper along with an explanation of the survey format 


and are encouraged to request clarification at any time. 


Once the survey is completed, the interviewees are asked to subjectively rate the 
success of the program, at the point in time the program is being evaluated, on a scale of 
zero to ten. A score of zero corresponds to complete and utter failure, no viable product 
produced and the program failed miserably. A score of ten corresponds to complete 
program success, the program produced an outstanding product, on time, within budget, 


with no bugs, and the customer was ecstatic with the quality of the product. To assist the. 
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interviewees in visualizing the scale the researcher draws and labels a scale as shown in 
Figure 2 and the interviewees are asked to place an “X” on the scale which represents the 
success of the program at the evaluation point in time. Once this is done, the researcher 


asks the interviewee for a specific numerical value to associate with the “X”’ on the scale. 


0123 45 6 7 8 9 10 


Total Perfect 
Failure | No Errors 


Figure 2. Overall Program Score Scale. 


Finally the researcher asks the interviewee for feedback on the survey instrument 
itself. For example, where there any questions which did not make sense? Are there 
words which the interviewee did not understand? Is the survey sistrancat too long? 
How would they improve the survey instrument? What was their overall impression of 
the survey instrument? This feedback is collected to obtain a feel for the level of 
frustration the survey induced in the interviewee and to provide glimpses into the 


viability of the survey instrument for future improvements. 


The email surveys include an additional survey document to collect background 
information such as the number of software developers working on the program, the time 
frame of the program and the overall program success score of the interviewee. The 
email interviewees are encouraged to contact the researcher via telephone or with 
questions. In one case, the email survey was followed up by a telephone call to gain 
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further feedback from the interviewee regarding the survey instrument. In this case the 
interviewee had very strong opinions about the survey instrument and wished to bring 
them to the researcher’s attention. In the other email surveys the interviewees respond to 


the additional background questions through email alone. 


The survey instruments utilized in both the personal interview surveys and the 
email surveys are identical in content. The only difference in the survey instruments is 
the medium through which they are applied. The personal interview QMM survey 
instruments are hard copies of the survey instrument which is emailed with the exception 
of the additional background-questions document and the data dictionary and frequently- 
asked-questions document. The emailed survey instruments are made up of the following 
files: 

e Microsoft Word file containing Part I of the QMM survey, the pair-choice 
questions. 

Microsoft Excel file containing Part II of the QMM survey, the yes/no questions. 

Microsoft Word file containing the additional background questions. 

Microsoft Word file containing background information on the QMM survey and 

the goals of the current research for the interviewee’s information. 


e Microsoft Word file containing a data dictionary of QMM terms and answers to 
questions frequently asked about the QMM survey instrument. 


Part I of the QMM survey instrument, the pair-choice questions, consists of two 
questions side by side on a single line with a column beside each question. The 
interviewee is asked to check the box next to the question or statement on each line which 
most cisely reflects what is or was happening on the program at the specific point in 
time (milestone, delivery, etc.) on the program which is being evaluated. The 


interviewee chooses between the two choices on each line of the survey. The statement 
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or answer which most closely reflects the program does not need to be an exact match. 
Each pair statement represents two differing ideas in an effort to ascertain a tendency of 
the individual in the area of interest, such as the use of formal requirements 
documentation versus informal or no documentation. Often the pair choices are repeated 
with different wording to confirm earlier choices and measure the strength of the 
tendency. The survey format, with the proper mix of questions and variation repetitions 
is intended to be used to reach consensus on issues and measure the strength of 


tendencies [Ref. 7]. 


Part II of the QMM survey instrument, the yes/no questions, consists of a single 
question per line with three columns next to it labeled “yes,” “no,” and “n/a” 
respectively. The interviewee answers the question as it pertains to the program manager 
and the program at the specific point in time the program is being evaluated by placing a 
check in the appropriate “yes”, “no”, or “n/a” box next to the question. The use of the 
“n/a” box is discouraged. However, it exists and is used in the cases in which the 
program manager does not have direct control over the issue which is asked in the 
question. For example, when an interviewee evaluates a government program manager 
who is in charge of a contractor developing the software, the government program 
manager does not have direct hire/fire authority over the development personnel. In the 
case in which the survey question asks if the program manager has direct hire/fire 
authority over the personnel, the appropriate answer would be “n/a”. The “n/a” answer 
column is present to allow the interviewee to indicate the program manager is constrained 
in the area being surveyed and does not penalize the program manager for factors beyond 


their control. 
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The weighting of the questions on the survey instrument are the same as used by 
Martin Machniak [Ref. 7]. The pair-choice questions in the requirements management 
section have a possible score of zero to two with different upper bounds on the score of 
each question based upon its relative weight and importance in the section. The possible 
scores in the estimation/planning management, people management, and risk 
management sections have a possible score of zero to one. The yes/no questions have a 
possible score of minus four to four with different upper and lower bounds on the score 


of each question based upon its relative weight and importance in the section. 


To score the results of Part I and Part II of the QMM survey instrument, the 
interviewee responses are entered into a Microsoft Excel spread sheet containing the 
QMM questions, response columns, and question weighting factors. A separate spread 
sheet file is created for each interviewee’s results for each part of the QMM survey. 
Thus, interviewee A - PM will have an Excel file containing the results for their QMM 
Part I results and a separate file containing their QMM Part II results. The spread sheets 
automatically calculate the total points for each of the four QMM sections: requirements 
management, estimation/planning management, people management, and _ risk 
management. ‘These results, along with the interviewee’s subjective overall program 
score, are entered manually ina QMM summary spread sheet for each interviewee which 
then calculates the total points for each QMM section, the overall QMM score, and the 


QMM percentage score. 


The summary data for each program and each interviewee is then manually 
entered into an Excel summary spread sheet. The data contained in the summary spread 


sheet identifies the program, the interviewee for the program, and the interviewee’s 
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scores for the following QMM sections: requirements management, estimation/planning 
management, people management, and risk management, QMM score, QMM percentage 
score, and overall program score. The data in the summary data spread sheets is then 


used for analyzing the data. 


B. ANALYSIS TECHNIQUES 


The data analysis consists of graphing the data to look for obvious trends, 
computing the mean, standard deviation, minimum and maximum values for the data 


range, variance, and various data set correlation to explore the data further. 


The QMM point totals for the four sections—requirements management, 
estimation/planning management, people management, and risk management—are 
converted into percentages and constrained to a scale of zero to ten by the following 
steps. The total points scored for each section are normalized to zero by subtracting the 
minimum possible points for each section from the points scored for the section. The 
resulting normalized score is then divided by total of the maximum point totals possible 
in each section minus the minimum possible points for each section to compute the 
percentage of the points scored in each section. These section score percentages are then 
multiplied by a factor of ten to convert the percentages from a scale of zero to 100 
percent to a scale of zero to ten. This is done to keep the scales of all the bar charts the 
same, providing for easier direct comparisons with the overall program scores which are 
already constrained to the scale of zero to ten. The equations are as follows: 


SECTIONmax - SECTIONmmn = SECTION pp 


SECTIONscorse — SECTIONmin = SECTIONScoreE NoRMALIZED 
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(SECTIONscore NorMALIZED/SECTION pp) = SECTIONpERcENTAGE SCORE 


SECTIONpercentace score * 10.0 = SECTIONscore scALep FROM 0 TO 10 


where 
SECTIONmax is the maximum number of points possible for the section 
SECTIONmwm is the minimum number of points possible for the section 
SECTION?pp is the total points possible for the section 
SECTIONscore NorMALIzep iS the total points scored for the section normalized to 
ZeIO 
SECTIONpercentacE score is the percentage of the points scored for the section 
SECTIONscorE scALED FROM 0 To 10 iS the percentage score for the section 
converted to a scale of zero to ten 
For graphing purposes and making direct comparisons with the overall program 
score, the QMM percentage score is divided by a factor of ten to constrain it to a scale of 
zero to ten. Thus, the summary data tables and bar charts, in Chapter TV and Appendix E 
respectively, report the QMM percentage scores on a scale of zero to ten with a score of 
zero corresponding to a QMM percentage score of zero percent and a score of ten 


corresponding to a QMM percentage score of 100 percent. 


The graphs created are bar charts displaying the values of all the surveyed 
programs for the following data sets: 


e Program manager’s (PM) overall program score and PM QMM percentage score 

e Individual development team member’s (IND) overall program score and IND 
QMM percentage score 

e PM overall program success score, PM QMM percentage score, IND overall 
program success score, and IND QMM percentage score 
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PM and IND overall program success scores 
PM and IND QMM percentage scores 

PM and IND requirements management scores 
PM and IND estimation/planning scores 

PM and IND people management scores 

PM and IND risk management scores 


The data analysis includes computation of the mean, minimum value, maximum 


value, standard deviation, and variance for the following data sets: 


PM overall program scores of all the programs 

PM QMM percentage scores of all the programs 

PM requirements management scores of all the programs 

PM estimation/planning management scores of all the programs 
PM people management scores of all the programs 

PM risk management scores of all the programs 

IND overall program scores of all the programs 

IND QMM percentage scores of all the programs 

IND requirements management scores of all the programs 

IND estimation/planning management scores of all the programs 
IND people management scores of all the programs 

IND risk management scores of all the programs 


The data analysis also computes the correlation between the following data sets: 


PM overall program scores and PM QMM percentage scores 

PM overall program scores and PM requirements management scores 
PM overall program scores and PM estimation/planning management 
PM overall program scores and PM people management scores 

PM overall program scores and PM risk management scores 

PM QMM percentage scores and PM requirements management scores 
PM QMM percentage scores and PM estimation/planning management 
PM QMM percentage scores and PM people management scores 

PM QMM percentage scores and PM risk management scores 

PM requirements management and PM estimation/planning management 
PM requirements management and PM people management scores 

PM requirements management and PM risk management scores 

PM estimation/planning management and PM people management scores 
PM estimation/planning management and PM risk management scores 
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PM people management and PM risk management scores 





IND overall program scores and IND QMM percentage scores 
IND overall program scores and IND requirements management scores 
IND overall program scores and IND estimation/planning management 
IND overall program scores and IND people management scores 

IND overall program scores and IND risk management scores 

IND QMM percentage scores and IND requirements management scores 
IND QMM percentage scores and IND estimation/planning management 
IND QMM percentage scores and IND people management scores 

IND QMM percentage scores and IND risk management scores 

IND requirements management and IND estimation/planning management 
IND requirements management and IND people management scores 

IND requirements management and IND risk management scores 

IND estimation/planning management and IND people management scores 
IND estimation/planning management and IND risk management scores 
IND people management and IND risk management scores 


PM overall program scores and IND overall program scores 

PM QMM percentage scores and IND QMM percentage scores 

PM requirements management scores and IND requirements management scores 
PM estimation/planning management scores and IND estimation/planning 
management scores 

PM people management scores and IND people management scores 

PM risk management scores and IND risk management scores 
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IV. RESULTS AND ANALYSIS 


A. RESEARCH BACKGROUND 


In order to encourage complete and open participation in the QMM survey, the 
interviewees are assured at the beginning of the survey process that the results of the 
surveys will be reported anonymously. To this end, the program data reported here is 
reported as program A, B, C, etc., and the letters associated with the individual programs 
do not reflect the order in which the programs were surveyed. Appendix B contains the 


responses to the survey for all of the participants. 


The ten programs surveyed range in size from a minimum of three software 
developers to a maximum of twenty-five software developers; see Figure 3. The time 
frame of the programs surveyed range from 1992 to present day, with most of them 
within the last two years; see Figure 4. All of the programs are DoD software 


development efforts. 


The minimum amount of time required to complete the survey instrument in a 
personal interview is an hour. The average time required to complete the survey 


instrument is two hours, and the longest time required is four hours. 
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B. GRAPH ANALYSIS 


To allow comparison between the different data types, a standard scale of zero to 


ten was chosen with the minimum, zero, corresponding to complete and utter program 
failure and zero percent of the total points possible for the QMM percentage score and 
the QMM section scores. The maximum possible score is ten which corresponds to 
complete program success and 100 percent of the total points possible for the QMM 


percentage score and the QMM sections. 
The QMM survey data is shown as bar charts in Appendix C, Figures C1 — C9. 


The bar charts are examined to determine if there are any obvious trends. The 
possibility of the program managers consistently rating the success of the program higher 
than the corresponding QMM percentage score is not found in the graphs, although it was 
expected. One instance of the program manager rating the program overall far higher 
than the QMM percentage score and the individual development team member overall 
program score and QMM percentage score is shown in program A; see Figure C3. A 
possible explanation for this is self-enhancement bias on the part of the program 


manager. 


An exciting result of the research is seen in the data for program F; see Figure C3. 
Program F does not utilize many of the currently accepted software engineering 
management processes and procedures such as formal risk management.. The program is 
ranked as highly successful by the software developers and the customers with an overall 
program success score of 9.0, yet the QMM percentage score consistently ranks the 


program as a 5.4. The exciting part of the discovery is that it appears the QMM survey 
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instrument does measure the implementation and the successfulness of the 
implementation of currently accepted software engineering management practices as 


defined by Martin Machniak [Ref. 7]. 


In programs C, I, and J, the program manager’s QMM percentage score is 
consistently higher than the program manager’s overall program success score and the 
scores of the individual development team member in the areas of overall program 
success score and QMM percentage score; see Figure C3. A possible explanation for this 
discrepancy is that the program manager believes he or she is implementing and using 
more of the software engineering management practices than are actually being 
implemented at the development-team level. If this is the case, then the QMM survey 
scores could be used to assist the program manager in identifying differences in 
perception between themselves and the rest of the development team, and take steps to 
improve the communication within the development team. Better communication could 
assist the program manager to better guide the program development team’s efforts and 
receive feedback on the processes and procedures which the program manager 


implements to enable the team to improve them. 


C. DATA ANALYSIS 


A summary of the program manager and independent development team member 
data for all the programs surveyed is presented in Table 3 and Table 4. In the tables, the 
QMM percentage score, requirements management, estimation/planning management, 
people management, and risk management scores are all normalized to a scale of zero to 


ten. A score of zero corresponds to zero percent or zero percent of the points possible for 
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the section. A score of ten corresponds to 100 percent or 100 percent of the points 


possible for the section. 


The correlations of the various data sets presented in the following tables are 
computed using the coefficient of correlation, p, where -1 <p <1. A value of p = -1 
implies a perfect straight line relationship between the correlated data sets with a negative 
slope. A value of p = 1 implies a perfect straight line relationship between the correlated 
data sets with a positive slope. A value of 9 = 0 implies no linear relationship between 


the correlated data sets. 


The data analysis for the program manager data is presented in Table 5. From this 
data it can be seen that there exists a relatively high divergence of 2.09 in the overall 
program scores for the various programs. This is expected as the programs all have 
different management, requirements, and work forces, and are constrained by different 


budgets and schedules. 





Table 3. Program Manager Summary Data. 
41 














IND | IND JIND Est. / 
Req. |Planning Ria 
Man. Man. 


4-32-82 $1 88 1 81 
[Bp | 8 | 87 [| 86 | 90 | 85 | 





Se a 


Table 4. Individual Development Team Member Summary Data. 
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Table 5. Program Manager Data Analysis. 


The standard deviation of the program managers in the area of people 


management is a relatively low value of 0.98 with a corresponding mean of 7.77. This 
indicates that the program managers believe they are good people managers or at least 


believe they are implementing many of the people management policies and procedures 
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the QMM survey instrument measures. This is further supported by the strong 
correlation between the program manager overall program score and the people 


management score of 0.771 in Table 6. 


The program manager estimation/planning management and risk management | 
sections have high standard deviations of 1.85 and 3.04, shown in Table 5, along with a 
strong correlation of 0.886 for these two sections, shown in Table 6. These management 
areas involve a great deal of uncertainty. The data indicates that program managers who 


do well managing the uncertainty in one area generally do well managing the uncertainty 
in the other area as well. 


PM /{PM Req.| PM Est./| PM 
QMM | Man. {Planning| People 
Score Man. 
Program 
i ld ll al 
Score 
a Oa 
El 
Planning 
i ad 
People 


Table 6. Program Manager Data Correlation. 











In Table 6, the areas of requirements management and risk management show the 
weakest correlation with the overall program score with correlations of 0.275 and 0.193. 


This indicates the implementation of the requirements management and risk management 
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practices do not play a big role in the subjective evaluation of the program manager on 


the success of the overall program. 


The program manager data correlations of the four QMM sections—requirements 
management, estimation/planning management, people management, and _ risk 
management—are strongly correlated with the program manager QMM percentage score. 
This is expected since the QMM percentage score is comprised of the four section scores. 
However, the strengths of the correlations are surprising. The strongest correlation is 
expected to be the correlation with the people management section as this section has the 
highest Importance Coefficient (IC) of 1.86. Yet, the people management section has the 
weakest correleation of the four sections with a correlation of 0.884 and the sections with 
the lowest IC values—estimation/planning management and risk management—have the 
strongest correlations of 0.933 and 0.922; see Table 6. These results indicate that the 
QMM percentage scores for the program managers are driven more by the scores in the 
estimation/planning management and risk management sections. This is not surprising 
given the fact that the program-manager standard deviation for the people-management 
section is low with a value of 0.98, and the mean for the section is relatively high with a 
value of 7.77. The people management scores for the various program managers are 
relatively constant. Therefore the factors which vary the most and distinguish the QMM 
percentage scores for the various program managers are the scores for the areas with the 
highest standard deviation, that is, the estimation/planning management and risk 


management sections. 





In general, the correlation analysis for the program manager data indicates that, 
except for the subjective overall program success score, all the variables have a strong 


correlation. 


Table 7 presents the results of the analysis of the data gathered from surveying the 
individual development team members. The standard deviation for the people 
management section is higher, 1.84, than the program manager standard deviation for the 
same section, 0.98. The mean for the individual development team member data is 6.27 
versus 7.77 for the program manager data. This indicates the individual development 
team members either do not think the program managers are as good at managing people 
as the program managers think they are or the individual development team members do 


not see or do not understand the people-management procedures and practices the 


program managers believe they have implemented on the program. 


IND IND [IND Est. / 
Moin QMM Req. |Planning Paci 
<i << e oe” Man. 


ee rae Pee eT ee eet 
al lll al al 
Deviation 


Variance| 2.67 | 268 | 223 | 5.51 | 3.37 | 7.30_ 





Table 7. Individual Development Team Member Data Analysis. 


The high standard deviations for estimation/planning management and risk 


management in Table 7 mirror the results for the program manager data analysis. 
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Perhaps this is due to these sections having more uncertainty involved in the management 


of them. 


The individual development team member data correlations; see Table 8, between 
the four QMM sections and the QMM percentage score, are all strong. However, the 
strongest correlation between the people management section and the QMM percentage 
score of 0.902, is expected based upon the IC coefficients for the four sections. What is 
unexpected is the requirements management section has a weaker correlation than either 


the estimation/planning management or risk management section. 


The strong correlation between the estimation/planning management and nsk 
management sections of 0.840 in Table 8 indicates that the program managers who are 


good at managing the uncertainty in one area are also good at managing the uncertainty 










in the other area. 
IND IND 
QMM | Req. |Planning| People 
- Score | Man. Man. 
Program | 
i Kd eli 
Score 
ia dll 
Man. 
Ree | [Pepe 
Plannine 7 
eae[ || 
People 


Table 8. Individual Development Team Member Data Correlation. 
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Table 9 presents the results of data correlations between the program manager 
data and the individual development team member data. The correlation of the QMM 
percentage scores for the program manager and the individual development team 


members is a strong 0.693. 


The correlation between the requirements management and risk management 
sections in Table 9 are strong: 0.769 and 0.686, respectively. This indicates that perhaps 
these two areas are the easiest to gain insight into for non-program managers on the 
practices and procedures implemented by the program manager. The lowest correlation 


is the people-management section. 


PM Req.| PM Est. / 
Man. |Planning 









PM PM 
ie QMM People 
Score Man. 
el ee 
Program 
w—9 A oe SE 
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Se ee ee ee 
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Table 9. Program Manager and Individual Development Team Member Data 
Correlation. 
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The data analysis and data correlations are performed on the research data without 
the inclusion of program F as this program does not use many of the recommended 
software engineering management practices such as formal risk assessment and 


management. The results of the analysis are presented in Tables 10 — 14. 


The exclusion of program F from the data set shows an increase in the strength of 
the correlation between the program manager’s overall program score and the QMM 
percentage score from 0.447 to 0.673 as shown in Table 11. The data for the program 
managers continues to show a very strong correlation between the overall program score 
and the people-management section. The high standard deviations for the 
estimation/planning management and risk management sections along with the strong 
correlation between these two sections also continues to be present; see Table 10 and 
Table 11. The program manager data continues to show strong correlations between all 


the variables as was seen in the previous data set; see Table 11. 


PM {PM Req.;jPMEst./; PM _  /|PM Risk 
ane QMM | Man. {Planning| People | Man. 
a ice Man. 


ie ates 
| High | | 8.93 | 
| Low 3.77 _| 
eg EEE 
Deviation 










Table 10. Program Manager Data Analysis Without Program F. 
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Table ]1. Program Manager Data Correlation Without Program F. 


The data for the individual development team members shows the same trends as 
in the previous data with the following exceptions. The correlation between the overall 
program score and the QMM percentage score has increased from 0.589 to a strong 


0.772; see Table 13. 


IND IND |IND Est./| IND 
QMM Req. /|Planning| People 
Score | Man. Man. 
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Standard| 1.53 1.70 1.57 2.33 2.77 
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Table 12. Individual Development Team Member Data Analysis Without Program 
F, 
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Table 13. Individual Development Team Member Data Correlation Without 

Program F. 
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Table 14. Program Manager and Individual Development Team Member Data 
Correlation Without Program F. 
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Additionally, the correlation between the overall program score and the areas of 
requirements management, estimation/planning management, and risk management have 


increased significantly to 0.708, 0.745, and 0.714 respectively; see Table 13. 


Since the program manager’s overall program score for program A appears to be 
unusually high, 9.9, possibly due to enhancement bias, the data analysis and data 
correlations have been computed without the inclusion of the program A and program F 


data. These results are presented in Tables 15 — 19. 


The program manager data continues to show the same trends as in the previous 
data analysis with the following exceptions. The overall program score now correlates 


very strongly with the four QMM sections; see Table 16. The overall program score 


PM PM (PM Regq.|/PM Est./; PM 
Program| QMM | Man. |Planning| People 
Score | Score Man. 


[Mean | 6.19 | 7.10 | 7.03 | 641 | 7.72 | 5.65 | 
[High | 8.00 | 893 | 877 | 9.34 | 881 | 9.46 | 
| Low | 3.00 | 4.84 | 477 | 3.77 | 6.06 ! 
Deviation 


1.05 
2.53 8 









2.9 










4 
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Table 15. Program Manager Data Analysis Without Program A and F. 
correlation with the QMM percentage score for the program manager has increased from 


0.447 in the original data set to 0.673 in the data set without program F to the present 


value of 0.891; see Table 16. This indicates the QMM survey instrument produces very 
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good QMM percentage score results. Additionally, all variables for the program manager 


correlate very strongly with each other. 


PM /|PM Req.|/PM Est./| PM _  /PM Risk 
QMM Planning|; People | Man. 
fate ene ton 
Program 
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Table 16. Program Manager Data Correlation Without Program A and F. 
Program 
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Table 17. Individual Development Team Member Data Analysis Without Program A 
and F. 
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Table 18. Individual Development Team Member Data Correlation Without 


Program A and F. 
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Table 19 Program Manager and Individual Development Team Member Data 


Correlation Without Program A and F. 
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The individual development team member data continues to show the same trends 
as in the previous data analysis with the following exceptions. The overall program score 
correlation with the QMM percentage score and the four QMM sections has increased in 
strength; see Table 18. The correlation of the overall program score with the QMM 
percentage score has increased from 0.589 in the original data set to 0.772 in the data set 
without program F data, to the present strong correlation value of 0.817; see Table 18. 
This indicates the QMM survey instrument is producing very good QMM percentage 


score results. 


Although the data correlations within the program manager data set and the 
individual development team member data sets are very strong, the correlation between 
the two data sets as shown in Table 19 remains weaker. The strongest correlation is 
between the program manager and individual development team member data for 
requirements management. Requirements management is an area which by necessity 
must be managed on the program to understand and clarify the scope and goals of the 
program. This area is then perhaps more visible to the entire development team whereas 
the other management areas do not have the same level of visibility and insight 


throughout the development team. 
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V. CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE 
WORK 


A. CONCLUSIONS 
The QMM survey instrument yields a strong correlation with the QMM 


percentage score and the overall program score for both the program manager and 
individual development team member data sets. This indicates the QMM survey 
instrument is a viable instrument for measuring the quality of management on a software- 
development program where the management policies and procedures are the same as or 
similar to the currently accepted software engineering management practices, as defined 
by Martin Machniak [Ref. 7]. In the case where the management practices and 
procedures are known to diverge from the currently accepted practices the QMM survey 
instrument may be used to measure the level to which the currently accepted practices are 
implemented but will yield a lower QMM percentage score and probably predict a lower 


success rate for the program than the actual rate. 


The QMM survey instrument may be useful in detecting discrepancies between 
the program manager’s and the individual team member’s perspectives and 
understanding. When the program manager’s QMM percentage score is consistently 
higher than their overall program score, the overall program success score and QMM 
percentage of the individual development team member, it may indicate the program 
manager believes he or she is implementing and successfully using more software 
engineering management practices on the program than are actually being implemented 


on the development team level. 


55 











The low standard deviation and high mean of the program managers in the area of 
people management indicates that the program managers believe they are good people 
managers. Further evidence of this is seen in the strong correlation of the program 


manager’s overall program score and the people management score. 


A higher standard deviation and lower mean in the area of people management 
from the individual development team member data than the program manager data 
indicates that the individual development team members do not believe the program 


managers are as good at managing people as the program managers themselves. 


The estimation/planning management and risk management sections consistently 
have high standard deviations in both the program manager and individual development 
team member data and strong correlation between these two sections as well. This 
indicates that there exists a sen deal of uncertainty in managing these areas and that 
program managers who are good at managing the uncertainty in one of these areas are 


usually good at managing the uncertainty in the other area as well. 


The correlations between QMM percentage score and the four QMM section 
scores is strong for both the program manager data and the individual development team 
data in all three data analysis phases. However, the correlation between the program 
manager and the individual development team member data is much weaker except in the 
area of requirements management. This indicates that perhaps the area of requirements 
management is more visible and better understood by the entire team than other areas of 


management. 
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The QMM percentage scores have a strong correlation with the overall program 
scores for both the program manager and the individual development team member data. 
This is particularly true for the data set which excludes the program A and F data. In this 
data set the correlation for the program manager is 0.891 and the correlation for the 
individual development team member is 0.817. This strong correlation indicates the 


QMM survey instrument is producing very good QMM percentage score results. 


B. RECOMMENDATIONS FOR FUTURE WORK 


The recommendations for future work include updating and further testing of the 
QMM survey instrument, developing a mechanism to provide the program managers with 
feedback on their management performance and possible courses of action to take to 
improve their performance, and integrating the QMM survey instrument results into 


existing cost, schedule and risk models to improve program estimation accuracy. 


Updating the QMM survey instrument questions includes updating the focus of 
the survey instrument, refining the question wording, and refining the question weighting. 
As advances and improvements are made in the area of software engineering 
management, the QMM survey instrument will need to be updated to reflect these 
advances and improvements. The wording of the questions should be further refined so 
the questions do not lead the interviewee in making their choice between two answers 
and so the clarity of the question intent is improved. Adjustment of the weighting of the 
questions within each QMM section and among the QMM sections is required to focus 
the QMM survey instrument on the areas which are the most important in determining 


and measuring the quality of management on a software-development program. Lastly, 
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reduce the total number of questions if it is determined that this is appropriate. Reducing 
the number of questions on the survey instrument includes examining tradeoffs related to 
the usefulness of the survey instrument versus the ease of executing the survey 


instrument if it is shorter in length. 


The QMM survey instrument requires more testing to formally verify and 
validate it. Unfortunately, most programs do not keep historical data which can be used 
to formally validate the QMM survey instrument. Data appropriate for a more formal 
validation of the QMM survey instrument includes schedule, cost, budget, requirements 
management, manpower, risk management, people management, and estimation/planning 
management data throughout the lifetime of a program or at least during the time period | 
when the program manager evaluated is in charge of the program. Formal verification 
and validation will require a long term commitment from the program and program 
manager to be evaluated. The researcher will need to collect the data at intervals to 
ensure the data is not lost as time passes. The QMM survey instrument should not be 
executed near the beginning of the data collection as it may induce a bias in the way the 
program manager manages. People concentrate their efforts in areas where they know 
they will be evaluated and giving the survey instrument to the program manager early in 
the program may alter their management style. This does not preclude gathering 
management quality data early in the data collection but a different survey instrument or 
technique should be employed which will not induce bias. One interviewee thanked the 
researcher for the opportunity to participate in the current QMM research as they found 


the QMM survey instrument questions to be a good reminder of all they should be doing 
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to manage the program. This sentiment was echoed by a number of interviewees as it 


reminded them of what they were not doing on the program. 


Provide feedback from the QMM survey instrument to the evaluated program 
managers to assist them in improving their performance. An intent of the QMM survey 
instrument is to apply it to the program manager at the beginning of or at some point 
during the execution of the program, give them feedback on their performance in each 
area, and then apply it to the program manager later in the program execution, or the 
execution of another program they are managing, to measure the improvements in 
management quality and provide further feedback. The QMM survey instrument 
currently does not provide specific feedback guidance for the program managers. In fact, 
the QMM survey instrument measures the performance of the program manager at a high 
level and only provides feedback to the program manager to the level of their score in 
each of the four QMM sections: requirements management, estimation/planning 
management, people management, and risk management. There is currently no provision 
for providing more specific feedback to the program manager other than in the area of 
people management which is made up of four sub-sections: human resources, 
communication, leadership and technical competency. Research is required to evaluate 
the specific questions in each section and determine which course or courses of action 
should be recommended for the program manager to pursue to improve their performance 
in the area depending upon their responses to the individual questions. The feedback can 
be in the form of an automated program which analyzes the results. However, each 
manager is different and their personal style and program situation is different. A better 


way might be to implement an automated program to rank the quality of management in 
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specific areas and to suggest possible courses of action with the intent of providing the 
information to a management trainer or mentor who would then work with the program 
manager to improve their performance in those areas. In the case where the program 
manager does not have access to a mentor or management trainer, the responses of the 
automated program could be combined with a Myers- 
Briggs personality profile of the program manager to suggest possible courses of action 
for the program manager to pursue and sources of information to help the program 


manager understand the suggested courses of action and the implications of the actions. 


The QMM survey instrument appears to detect differences in perception between 
the program manager and the development team on what practices and procedures the 
program manager believes are implemented and working and the actual state of 
understanding and implementation of the practices and procedures on the development 
team level. The program manager feedback mechanism described above could include 
detection of areas where the perceptions of the program manager and the development 
team differ greatly and the information could be used to alert the program manager and/or 
the program manager’s management trainer or mentor to the problem. Knowledge of 
areas of potential misunderstandings enables the program manager to begin working on 
opening up the communication channels to better guide the development team’s efforts 
and receive feedback on the processes and procedures which the program manager 


implements, enabling them to improve them. 


The QMM survey instrument results could be used as an input into current 
program cost, risk, and schedule estimation models to improve the resultant estimations. 


Currently these models do not incorporate the quality of software-development 
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management as a factor other than to assume good management. As this assumption is 
not necessarily a good assumption, using the QMM results input for the models may 
increase the accuracy of the estimation results. Barry Boehm [Ref. 2] states the 
COCOMO model assumes good management for two reasons: the quality of management 
is difficult to measure, and providing a poor manager with more resources to complete a 
program than a good manager is bad management practice. Since the COCOMO model 
assumes good management and provides estimates for managers based upon this 
assumption, the model will estimate costs and schedules for good managers. This gives 
the poor managers the same guidelines as the good managers. It is human nature to live 
up to or down to expectations. If the model provides estimates based upon bad 
management, will.this lower the expectations and goals of the manager and produce 
programs with lower success levels? It is an interesting question. For planning purposes, 
accurate estimates are necessary. However, what will the cost of accurate estimates be in 
the case of poor management, assuming accurate estimates are desired? The research 
involves implementing a QMM survey score as an input into the current program 
estimation models to provide “accurate” data on the quality of the program management, 


such as with a formal model of risk assessment in a software program; for example, see 


[Ref. 8]. 
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APPENDIX A. QUALITY MANAGEMENT METRIC SURVEY 
FORMS WITH SCORING 


Quality Management Metric survey instrument forms template with scoring. 
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Program Name YES-NO-N/A Questionnaire Scoring Template Date 
No. Requirements Management Questionnaire Yes No N/A 


| _14{PM chose to have a formal requirements list f4fTolo, 
| 21Requirements recorded in some wa Lor aisle 


Written requirements were part of some formal document f41{/of 0. 
| 4|Writtenrequirementswereinformal ss —“—*s*s—“‘—s—‘“‘“<~w CECT CO 
| 5|Atleastsome requirements were oralonly ss —“—‘s*~s*s*s~*~Ss~éssSCC*zrCSCzLC«C 
| 6lAlistakeholderswereidentified == —“‘~‘;*™*™*~*™C”C*dCh~dE=TC”T COT 
|__7All stakeholders participated in the requirements extraction sd] 2 | CO] OL 


| 81Some stakeholders participated in the requirements extraction Ee a AO |, 0 


| _9|Management extracted requirements, no stakeholder involvement 


Management passed requirements to development team hae 20 |. | 


| 11|Stakeholders not involved in Management extraction, butapproves sd -1| O | O| 
Management gets inputs from stakeholders, then develops requirments REEsee 
|_13|Developers work informally with users to arrive at requirements _—s———s«d(|:' 1 ‘| (0 [ 0 | 
alae as 19, but management oversees and formalizes | »Lo Lo] 


lf a waterfall or sequential developrnent strateqy: 


All requirements complete before design ha SO. | 


ale 050: 
_17|Requirements informal priortodesigneffot_ CE TT OT OO 
_18|Requirementsserveasinput Ct 
|_19/Length of time for requirements work greater than development work | 2 | -1 | 0 


| 20/Requirements developedinparalleltodesign ss —“‘S;OCCd iC C*diT CO! 
OR_/fa prototype, throwaway, or other development strategy: 

| 15|/Learn about requirements through developmentefforts ss ™~—<“‘d; CT Cd CO 
| 16|No coding untilallrequirementsaredefined ss —“‘;éOO! C*L OCLC CL CO 
Requirements formal prior to design effort 4} 0] 0. 
| 18|Requirementsserveasoutput — ——“*‘“‘“‘;*™*™C*é‘édrC CAACsdC= ACE‘ 
Requirements definition work in parallel to development efforts 2 | -4 1-0 | 
P14 -1] 0 | 
| 22|\Changemanagementexiss—“‘“‘S™SOCOS;™;™;!;!!OC(C*dY Cd COO 
| 23|Changemanagementisformal = —“—*~*™*™~*~*~*~™C~C”C”C”CSCSdCSCEC COT 
Project strategy is consistent throughout development 11010. 
| 25|Requirementsareupdated i —itsws~—“‘“‘(‘“‘S;TOO (Cd Crd Or CO 
| 26|Configuration Management (CM)exists——“—*é‘—s*s*s—s—s—s—s—s—s—s—séC ST YC 




















| 27|CMisformal ——“‘“(C;S*S*S™C™~COCCCOSS AF TC 
| 28|Requirementsaretestable  — —“‘“C;éO™SOSO!UtCdL QUT QI COO] 
| 29|Requirements testing considered/implemented during extraction sss] 2 [ Of 0, 
Requirements testing plan exists 27010) 
Requirements testing is formal 11010. 
All requirements have priorities eae aes 
All requirements must be implemented ro}; 1]o| 
Requirements are tested bt a 8 
All requirements are equally important Ot}; 1] 0. 
At least some requirements have priorities | 11] ofo| 
| 37|Allrequirementsaretraceable = —(‘“S;éOUlUdL ACT [OO] 
| 38[Traceabilitynotimportant ss —“(‘“S;OF€F€Urdr rE tif (cl 
| 39/Eachrequirementhasanauthor ss — s—“‘“CSOCCOC(C*dC AGWéiP COT CO 
| 40}Who authored requirement is not important O11] 0. 
Initial set of requirements to be implemented, no requirements creep 1 o}| 1] 0) 
| 42|Structured andtracked changestorequirementsonly ——“‘“‘é‘sYL ‘iT -t'*if:=t‘ 
|_43|Change is inevitable, changes allowedatalltimes ss ——“=;‘*~*sé~srCTC*dECCST:C 
| 44|Changeisinevitable, butchangeslimited ——“—*~—s—s—s—s—s—“‘“‘“‘“aC CL C*iL OY CO 
| 45|Requirementscontrolfunding  — ——s—“—*i‘“s*s—“‘“‘“‘“SNCC#*;;*nSHCzr COC*L‘CO 
Requirements history kept 14] -1] 04 
Baseline established for requirements at some point prior to develop 12] 2] 0] 
TOTALSCORING | [| jf | 


Enter total score on QMM score sheet block e. 
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Program Name YES-NO-N/A Questionnaire Scoring Template - Date. 


No. Estimation/Planning Questionnaire Yes No NA 
1]/A volume product metric used (LOC, # of files, # of screens, pages of doc 14} of 0] 
| 2|Measure used for various product elements (modules, components, CSCl)__|[ 1 | 0 | 0_ 


| 31Product measures made by phase (amt at implementation, LOC changed at unit test) 14] of 0] 
[4] Other product attributes measured (FP, throughput, mem cap, cyclomatic complexity) | 1 | 0] 0 | 


Product metrics tracked and updated throughout program execution 
Event count process metric used (# defects in test, reqmt changes, milestones met) 
Time measure process metric used (cycle time 
Process metrics tracked and updated throughout program execution 
Program cost estimations made from product or process metrics 
Program cost estimations tracked and updated to reflect progress/changes 
Factor analysis performed on program 
Program's primary purpose, including major functions and deliverables known 
Work breakdown structure developed 
Task estimated with realistic expectations of productivity probabilities 
Schedules developed based on realistic expectations 
Schedules tracked and updated based on new information 
17|Detailed activity lists used for clearly defined completed/not completed tasks 
ity assurance plan or similar to aid in detecting defects early in program 
COCOMO estimates performed 
CSCI clearly defined and tasked 
Estimates completed ad hoc 
Gantt charts used and updated 
Resource estimations (working hrs, job categories, task activities) done 
arned value established 
25|Eatned value tracked throughout program 
Quality expectations established for product with users and stakeholders 
27| Critical path for program tasks developed and tracked 
Meaure of effectiveness (MOE) or Figure of merit established and tracked 
Estimates are updated routine 
Schedules are updated routine! 
31|Estimations are made by program management (top-down 
Estimations are made by program team members (bottom-up 
Automated program tracking used 
thorough in tracking and reporting schedules and financials 
BS developed only as data call, not used in planning 
arned value used to track program progress 
M insists on prioritizing work reduction as schedule/funding compromised by stakeholders 
stimations are done using both top down and bottoms up approaches 
involved in planning process 
Hardware also considered in estimation process 
iled 
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uests estimated individuall 
art of each team member's responsibilities 
PM dictates schedules to program team 
i nlanned in schedule 
Defined tangible milestones established for program tasks 
7\Test planning done at the start of the program 
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Estimations are completed by those performing the tasks 

Sensitivity analysis performed for program choices 

Software deployment planning completed | 4] -1] 0/1 
TOTALSCORING] | {| | | 


Enter total score on QMM score sheet block f. 
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Program Name YES-NO-N/A Questionnaire Scoring Template Date. 


No. People Management Questionnaire Yes No NA 


|_1|PMis accessable inpersonbyeachteammember CE tT OO CO 
| _2|PM is accessable viaemailbyeachteammember CE CT OO OT 
|_S|PMis accessable viaphonebyeachteammember Et | CO OT 
|_4|PM not only considers a person’s suitability, not also desire tobe onateam | 1 | 0 | 0 | 


| _5{PM consults with each team member regarding their career goals A O10 | 
| _61PM regularly holds meetings to inform team of program progress | 21-41] 0) 
PM solicits opinions from team members before making decisions | 2]|-11 0) 


2|PM prioritizes problems or conflicts within the program 
PM empowers program members to recommend firinas of other members 


PM specifically assigns work to each program member 


PM sets communication protocol to be followed 
PM readily makes tough decisions 


PM takes control in difficult/ problem areas 

M looks ahead to new programs, new upgrades of existing program 
|PM maintains regular communications with allstakeholders 
PM maintains regular communications with users 
PM encourages program team communication with users Ra 
PM encourages program team communication with stakeholders 


PM facilitates horizontal communication within program | 
|PM facilitates communication duringintegration —CCsidS=Cd CS 
PM holds meetings without clear objectives listed prior to meeting | 94 | 
|PM must approve all decisions withinthe program Cd -T 
PM must approve allinteractions withstakeholders 
PM must approve allinteractions withusers 


it 

2|PM makes all presentations to stakeholders/users 30s. 
PM is considered "flexible" in terms of program members personal issues ae 
ees 


io) 
ofades ‘ efegde tT ' 


cs 


PM, at least occasionally, schedules/promotes outside work team activities 1] 0. 


PM is readily willing to listen to program problems and complaints 





alelalelelsleslels slvlsistlsialallelalelalelslslole 
Ol — OOl™ NOLO lO fOOI~I | | Ot] iGo © 


| 36|PM takes action to resolve program problems and complains | 1.:'| -1'[ 0 | 
|_37|PM is generally respected by stakeholders, users, andorganization —s——s| 1'[ -1[ 0_ 
| _38/PM sometimes fails to grasp important technical issues inprogram —s——si[| -1: | 1:*[ (OO | 
|_39|PM recruits program team members from outside organization Ss | 1:'[ -1'[ (O| 
|_40|PM directs whatneedstobedoneanddirectshowtodoit == | -1 [1 [ 0 | 
Program personnel have clearly defined specific tasks Oo}; 1] 07 
Although individual's tasks are specific, each exposed to the "bigger picture" | 2 | -1| 0— 





PM has clearly defined his/her expectations for each individual -2|-1{ 0| 
PM delegation of duties is usually seemless in execution (1]/ 0}; 0 
PM acts as facilitator to solving personnel conflicts (2) -1{ 0) 


PM attempts to motivate individuals on the program team | 2] -1] 0) 
PM clearly separates technical from managerial roles for individuals Oo; 1{o- 
PM directs how he/she expects the task to be accomplished | oO} 1] 0) 


PM directs what needs to be done, but does not direct how | 2] -1] 0] 
| 50}PM attempts to spotlight individuals in the program for positive exposure | 21-1] 0] 
TOTALSCORING] | [| Jf | 


Enter total score on QMM score sheet block g. 
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Program Name YES-NO-N/A Questionnaire Scoring Template Date_ 


No. Risk Management Questionnaire Yes No N/A 
Risk Management (RM) is specifically an activity in the program | 4] -4] 0| 
|_2|RMisformalanddocumented ss a C“‘“(‘‘O™OC*#*#C#*#C#C#C#*drSCS:«dC:« 3] 
ecific RM plan exists | -2| 0 | 
RM is required in the program, but not used during the program 
RM is done prior to the program execution 
RM is done by an outside entity to the development 
is done internally onl 
RM is both internally performed and externally assessed 
olanning occurs during or after major milestones in the program 
Risk Assessment is only a management function 
RM is informal or non existent 
here is a RM plan, but it is not updated or tracked 
isks are only generalized 
Each risk is delineated 
Each risk has a consequence 
Each risk has a likelihood rating of some sort 
Each risk has a mitigation strateq 
Risk Management is automated 
Risks are tracked 
ret analysis performed 
RM drives decisions in the program 
Risks have probabilities 
Risk Management is ad hoc 
RM information is shared with all stakeholders (as appropriate 
Risks are weighed relative to other program risks 
j program team activi 
prior to program start 
personnel risk 
ends on human decisions 
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t includes technology risks 
program manager 
Risk Assessment includes requirements risks 
Risk Assessment includes user risks (too little involvement of user 
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i ration risks 
ing requirements change (feature creep 
1|Risk Assessment includes dependent projects/programs risks 

2| Documentation proof exists to demonstrate following risk management plan 
High risk have measured tracking (high profile status 
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nternal organizational checklists used for risk assessment 
7 
Risk Assessment includes internal organization risks 
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TOTAL SCORINGE [TL 


Enter total score on QMM score sheet block h. 
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Program: QMM Summary Score Sheet 





QMMSCORE  _—_—__ 000 SCORE 


Max. QMM score possible 528.00 
Min. QMM Score possible -130.86 
QMM percentage score 19.86% 


Objective/Subjective view of the overall success of program on a scale of 0 to 10 
(0 being total failure, 10 being perfect program total success) 

Program: 

Success Score: 
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APPENDIX B. RESEARCH DATA 


This appendix contains the survey responses to each question for all survey 
participants. The participants are identified by the letter of the program with a dash 
followed by a P for the program manager and a number for the individual development 
team members. Each table has the corresponding QMM survey section title at the top 
and the number of the question is in the left most column. Individual responses to the 
questions are indicated with a “1” in the appropriate box. The abbreviations used in the 


tables are the following: 


Lt. - indicates the answer selected is the left-hand answer on the pair-choice 


survey row 
Rt. - indicates the answer selected is the right-hand answer on the pair-choice 
survey row _ 
yes - indicates the answer selected is the yes column on the yes/no survey 
no - indicates the answer selected is the no column on the yes/no survey 
n/a - indicates the answer selected is the not applicable column on the yes/no 


survey 
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APPENDIX C. BAR CHARTS OF THE RESEARCH DATA 


This appendix contains bar charts of the research data. 














0 = Total Failure 
10 = Total Success 


Program Level of Success 


PM Program and QMM Scores 


EJ PM Program Score 


@PM QMM Score 





nnn 
patetetofete ta 


EI PMProgram Score, 9.9 | 75 | 8 | 7 | 


MPM QMM Score 


Figure Cl. 





Program Manager Overall Program Scores and QMM Percentage Scores 


112 











IND Program and QMM Scores 











ese ematical a 
Hehe eee eee See eee 
ee Rane eee 











nn 

“a 

8 

$23 

O35 0 

ss S 

° 

s Lv EIIND Program Score 
Cc = 

3 = = BIND QMM Score 

& 1 | | 

Soo 

te, bd 

Le] 

_™ 

oO. 


Figure C2. Individual Development Team Member Overall Program Scores and 
QMM Percentage Scores 
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Figure C7. | Program Manager and Individual Development Team Member 
Estimation/Planning Management Scores 


118 











Scores 
0 = 0% of Possible Score 


10 = 100% of Possible Score 


ame EEE 
BIND PeopleMan.| 56 | 85 | 78 | 41 | 65 | 66 | 58 | a1 | 79 | 25 | 57 | 


Figure C8. 








People Management Scores 


Fie Ca GF 
ie Oe ee 2 a FR DO 
et __| T 


£3 PM People Man. 


MIND People Man. 





Program Manager and Individual Development Team Member People 
Management Scores 


119 








Risk Management Scores 


PM Risk Man. 





MIND Risk Man. 


100% of Possibie Score 


Score 
0= 0% of Possible Score 


10 








mn HE ns hh no 

Fae Sa 
EIPMRiskMan.| 40 | 95 | 5.1 | 

| 56 


C 
WIND Risk Man.| 5.1 | 9.0 | 17 






Program 
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Management Scores 
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